Image Processing Apparatus and Camera System

ABSTRACT

An image processing apparatus or camera system comprises an image sensor  1,  a geometrical position calculation device  6  for performing predetermined correction of a distortion, a first address table  10  for storing information correlating an input side address based on the calculation results of the geometrical position calculation device  6  to an output side address as a reference, a sort unit  11  for sorting the output side addresses according to the input side addresses, a second address table  12  for storing information correlating the output side address to the sorted input side address as a reference, and an address matching device  13  for matching the input side address of input side image data DI with the input side address stored in the second address table  12  and outputting output side image data DO.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of co-pending U.S. application Ser.No. 12/669,750 filed Jan. 19, 2010, which is a national phase filing ofPCT/JP2008/062984 filed Jul. 18, 2008, which claims priority to JapaneseApplication No. 2007-189807 filed Jul. 20, 2007, all of which areincorporated by reference herein in their entireties.

TECHNICAL FIELD

This invention relates to an image processing apparatus and a camerasystem. More specifically, the invention relates to those useful whenapplied to an image processing apparatus or an omnidirectional camerasystem which perform correction of a distortion when doingomnidirectional monitoring through a fish-eye lens, or an imageprocessing apparatus or a camera system which perform processing fordistorting a captured image.

BACKGROUND ART

A fish-eye lens is known as a lens having a wide-angle field of view orvisual field. A proposal has been made for an omnidirectional camerasystem for monitoring a predetermined region with the use of thisfish-eye lens. Such an omnidirectional camera system generally hasfunctions such as panning, tilting, zooming (i.e., pan, tilt, zoom) androtation (or rolling or roll). In recent years, a proposal has also beenmade for an omnidirectional camera system which, as compared with theconventional mechanical pan, tilt or zoom mode, electrically processesinput side image data captured by the fish-eye lens, thereby fulfillingvarious functions, such as pan, tilt, zoom and rotation, and eliminatingthe distortion of the image, without moving the apparatus.

FIG. 14 is a block diagram of this type of omnidirectional camera systemaccording to the conventional technology. As shown in this drawing, aCMOS image sensor 1, which is an imaging means, forms image data as anelectrical signal based on a wide-angle image captured by a fish-eyelens 2. This image data is an electrical digital signal having adistortion component remaining as such when captured by the fish-eyelens 2. This signal is subjected to predetermined color processing at acolor processing unit 3, whereby it is written into a frame memory 4 asinput side image data DI having color information as well as luminanceor brightness information.

A parameter setting device 5, on the other hand, has settings ofparameters concerned with pan, tilt, zoom and rotation for cutting out aregion to be displayed, in the image captured by the fish-eye lens 2,and parameters related to the center and radius of the fish-eye lens 2.That is, the parameter setting device 5 functions as an input/outputdevice for these parameters. The parameters about the pan, tilt androtation are set as angular information, the parameter about the zoom isset as magnification information, and the parameters about the centerand radius of the fish-eye lens 2 are set as positional information andnumerical information. A geometrical position calculation device 6calculates the geometrical position of each pixel in the input sideimage data DI, which corresponds to each pixel on a display screen(output screen) 9 a of a display device 9, in order to correct thedistortion, by the fish-eye lens 2, of the image in the region to be cutout as output based on the parameters set in the parameter settingdevice 5.

An output side image data generation unit 7 forms output side image dataDO corrected for the distortion based on the geometrical position ofeach pixel in the input side image data DI calculated by the geometricalposition calculation device 6, and the input side image data DI storedin the frame memory 4. This output side image data DO is data obtainedby sequentially combining brightness information, etc. based on theinput side image data DI for each pixel on the output screencorresponding to the geometrical position. The output side image data DOis sequentially stored into a buffer memory 8, pixel by pixel, and isreproduced on the display screen (output screen) 9 a of the displaydevice 9 frame by frame. In this manner, the image in the predeterminedcut-out region and corrected for distortion is reproduced on the displayscreen 9 a.

As a publication which discloses a technology of the same type as thatof the above-described conventional technology, the following patentdocument 1 is existent:

Patent Document 1: JP-A-2000-83242

With the above-described omnidirectional camera system, the input sideimage data DI, which is the output signal of the image sensor 1 as theimaging means, is written into the frame memory 4. After the writing ofthe input side image data DI corresponding to one frame is completed,the output side image data DO is formed by reference to the contents ofstorage in the frame memory 4. Thus, during such a series of processingsteps, a time lag occurs. Such a time lag manifests itself as a displaydelay on the display screen 9 a.

The present invention has been accomplished in the light of theabove-described conventional technologies. It is an object of theinvention to provide a camera system capable of achieving a series ofprocessings, without using a frame memory, in the conversion of inputside image data into output side image data, which involvespredetermined processing of a geometrical position, such as correctionof distortion of the input side image data.

SUMMARY OF THE INVENTION

A first aspect of the present invention for attaining the above objectis an image processing apparatus, comprising:

parameter setting means which has a setting of a parameter concernedwith at least one of pan, tilt, zoom and rotation for cutting out aregion to be displayed in an image taken in by a lens;

geometrical position calculation means for calculating a geometricalposition of each pixel in input side image data based on an outputsignal of imaging means, the each pixel corresponding to each pixel onan output screen, in order to perform predetermined transformation of ageometrical position of an image in the region based on the parameter;

an address table for storing table information which is combinedinformation obtained by correlating an input side address, as an addressof the each pixel of the input side image data based on calculationresults of the geometrical position calculation means, to an output sideaddress as a reference which is an address of the each pixel on theoutput screen; and

address matching means which checks the input side address of the eachpixel of the input side image data, loaded in real time, against theinput side address stored in the address table, and when both input sideaddresses are coincident, combines the input side image data at theinput side address with the corresponding output side address to formoutput side image data, and also sends out the output side image data.

A second aspect of the present invention is a camera system, comprising:

imaging means for forming a digital signal representing an image takenin by a lens;

parameter setting means which has a setting of a parameter concernedwith at least one of pan, tilt, zoom and rotation for cutting out aregion to be displayed in the image;

geometrical position calculation means for calculating a geometricalposition of each pixel in input side image data, the each pixelcorresponding to each pixel on an output screen, in order to performpredetermined transformation of a geometrical position of an image inthe region based on the parameter;

an address table for storing table information which is combinedinformation obtained by correlating an input side address, as an addressof the each pixel of the input side image data based on calculationresults of the geometrical position calculation means, to an output sideaddress as a reference which is an address of the each pixel on theoutput screen; and

address matching means which checks the input side address of the eachpixel of the input side image data, loaded in real time, against theinput side address stored in the address table, and when both input sideaddresses are coincident, combines the input side image data at theinput side address with the corresponding output side address to formoutput side image data, and also sends out the output side image data.

A third aspect of the present invention is the camera system accordingto the second aspect, wherein

the address table stores table information in which the output sideaddresses in the table information based on the calculation results ofthe geometrical position calculation means in connection with a specificregion are rearranged according to the input side addresses.

A fourth aspect of the present invention is the camera system accordingto the third aspect, wherein

the address table stores a plurality of pieces of table information inwhich the output side addresses are rearranged according to the inputside addresses in connection with a plurality of the specific regions.

A fifth aspect of the present invention is a camera system, comprising:

imaging means for forming a digital signal representing an image takenin by a lens;

parameter setting means which has a setting of a parameter concernedwith at least one of pan, tilt, zoom and rotation for cutting out aregion to be displayed in the image;

geometrical position calculation means for calculating a geometricalposition of each pixel in input side image data, the each pixelcorresponding to each pixel on an output screen, in order to performpredetermined transformation of a geometrical position of an image inthe region based on the parameter;

an address table for storing table information which is combinedinformation obtained by correlating an input side address, as an addressof the each pixel of the input side image data based on calculationresults of the geometrical position calculation means, to an output sideaddress as a reference which is an address of the each pixel on theoutput screen;

matching sort means for rearranging the output side addresses stored inthe address table according to the input side addresses;

a matching address table for storing table information which is combinedinformation obtained by correlating the output side addresses to theinput side addresses upon rearrangement by the matching sort means; and

address matching means which checks the input side address of the eachpixel of the input side image data, loaded in real time, against theinput side address stored in the matching address table, and when bothinput side addresses are coincident, combines the input side image dataat the input side address with the corresponding output side address toform output side image data, and also sends out the output side imagedata.

A sixth aspect of the present invention is the camera system accordingto any one of the second to fifth aspects, wherein

the geometrical position calculation means finds the input side addressto decimal places, and outputs the input side address as a decimalvalue, and

the address matching means uses, as a reference, a specific pixelcorresponding to a whole number part of the input side address in theinput side image data, finds brightness of the specific pixel byinterpolation for weighting with a value of a fractional part of theinput side address based on brightness of a plurality of pixels adjacentto the specific pixel, and takes the brightness of the specific pixel asbrightness of a pixel in the output side image data corresponding to thespecific pixel.

A seventh aspect of the present invention is the camera system accordingto the sixth aspect, wherein

the address matching means has a buffer memory for storing at least oneline equivalent of data, and is adapted to find the brightness of thespecific pixel belonging to a next line by use of brightness informationon each pixel stored in the buffer memory.

An eighth aspect of the present invention is the camera system accordingto the second to seventh aspects,

further comprising a buffer memory for storing the output side imagedata,

the buffer memory having the output side image data written randomlythereinto, with the output side address as a reference, each timeresults of the checking are coincident, and

wherein readout of the output side image data is performed sequentially.

A ninth aspect of the present invention is the camera system accordingto the second to seventh aspects,

further comprising a buffer memory for storing the output side imagedata,

the buffer memory having the output side image data written sequentiallythereinto each time results of the checking are coincident,

further comprising read-out sort means for rearranging memory addressesof the buffer memory according to the output side addresses, and aread-out address table for storing table information which is combinedinformation obtained by correlating the memory addresses to the outputside addresses as a reference upon rearrangement by the read-out sortmeans, and

wherein readout of the output side image data is performed randomlybased on the table information of the read-out address table.

A tenth aspect of the present invention is the camera system accordingto any one of the second to ninth aspects, wherein

the lens is a fish-eye lens having a wide-angle visual field, and

the transformation in the geometrical position calculation means isprocessing for correcting distortion of the image in the region.

An eleventh aspect of the present invention is the camera systemaccording to any one of the second to ninth aspects, wherein

the transformation in the geometrical position calculation means isprocessing for distorting the image in the region.

According to the present invention, the input side address of each pixelof the input side image data, which has been loaded in real time,against the input side address stored in the address table. When bothinput side addresses are coincident, the input side image data at theinput side address is combined with the corresponding output sideaddress to form output side image data. Thus, there is no need for aframe memory as in the conventional technology, which stores the oneframe equivalent of input side image data as output signals from imagingmeans.

Consequently, a delay due to the time for writing the one frameequivalent of image data into the frame memory can be eliminated, andthe image taken in can be displayed promptly as a reproduced image. Sucheffects are remarkable, particularly, with a monitoring camera system orthe like which targets a moving body as an object of imaging.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an omnidirectional camera systemaccording to a first embodiment of the present invention;

FIG. 2 is an explanation drawing conceptually showing the concreteconfiguration of a parameter setting device shown in FIG. 1;

FIG. 3 is an explanation drawing conceptually showing display modesbased on the method of installing a fish-eye lens shown in FIG. 1;

FIG. 4 is an explanation drawing for illustrating the principle ofcorrection of a distortion in a geometrical position calculation deviceshown in FIG. 1;

FIGS. 5( a) to 5(e) are explanation drawings for illustrating theprocedural steps of a processing operation in the geometrical positioncalculation device shown in FIG. 1;

FIGS. 6( a) to 6(c) are explanation drawings for illustrating imageheight characteristics in the fish-eye lens shown in FIG. 1;

FIG. 7 is an explanation drawing conceptually showing the relationshipamong the geometrical position calculation device, an address table, asort unit, and a matching address table shown in FIG. 1;

FIGS. 8( a) to 8(g) are explanation drawings conceptually showingsorting in the sort unit shown in FIG. 1;

FIG. 9 is an explanation drawing conceptually showing the hardwareconfiguration of a multi-bit processing mode for realizing sorting inthe sort unit shown in FIG. 1;

FIGS. 10( a) and 10(b) are explanation drawings conceptually showing thedetailed configuration of an address matching device and a buffer memoryshown in FIG. 1;

FIG. 11 is a block diagram showing an omnidirectional camera systemaccording to a second embodiment of the present invention;

FIG. 12 is an explanation drawing conceptually showing the relationshipamong a geometrical position calculation device, an address table, asort unit, and a matching address table shown in FIG. 11;

FIGS. 13( a) and 13(b) are explanation drawings conceptually showing thefunctions of an address matching device and a buffer memory shown inFIG. 11; and

FIG. 14 is a block diagram showing an omnidirectional camera systemaccording to a conventional technology.

Embodiments of the present invention will now be described in detailbased on the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram showing an omnidirectional camera systemaccording to a first embodiment of the present invention. In thisdrawing, the same portions as those in FIG. 14 are assigned the samenumerals as those in FIG. 14. As shown in FIG. 1, a CMOS image sensor 1,which is an imaging means, forms image data as an electrical signalbased on a wide-angle image captured by a fish-eye lens 2. This imagedata is an electrical digital signal having a distortion componentremaining as such when captured by the fish-eye lens 2. This signal issubjected to predetermined color processing at a color processing unit3. The color processing unit 3 forms input side image data DI in pixelsfrom the output signal of the image sensor 1 in a layered arrangement.As a result, an address matching device 13 is supplied with the inputside image data DI which is a combination of brightness information andcolor information per pixel.

A parameter setting device 5 has settings of parameters concerned withpan, tilt, zoom and rotation for cutting out a region to be displayed,in the image captured by the fish-eye lens 2, and parameters related tothe center and radius of the fish-eye lens 2. The parameters about thepan, tilt and rotation are set as angular information, the parameterabout the zoom is set as magnification information, and the parametersabout the center and radius of the fish-eye lens 2 are set as positionalinformation and numerical information.

FIG. 2 is an explanation drawing conceptually showing the concreteconfiguration of the parameter setting device 5 which functions as aninput/output device in this case. As shown in the drawing, theparameters on the pan and tilt are set as angular information byoperating operation buttons 5 a, 5 b, 5 c, 5 d. The parameter on thezoom is set by operating operation buttons 5 e, 5 f.

A mode selector button 5 g selects a mode based on a difference in themethod of installing the fish-eye lens 2. In more detail, as shown inFIG. 3, there are a circling display mode and an orthogonal displaymode, depending on the method of installing the fish-eye lens 2. Thecircling display mode is classified as a circling display upward mode(desktop or the like) and a circling display downward mode(ceiling-mounted), and takes a picture, mainly, of the surroundings ofthe fish-eye lens 2. The orthogonal display mode is designed for takinga picture, with the optical axis being directed in the horizontaldirection as with an ordinary camera. By selecting any of the respectivemodes, the method of coordinate development for the input to thefish-eye lens 2 can be changed for display.

A geometrical position calculation device 6 shown in FIG. 1 calculatesthe geometrical position of each pixel in the input side image data DI,which corresponds to each pixel on a display screen (output screen) 9 aof a display device 9, in order to correct the distortion, by thefish-eye lens 2, of the image in the region to be cut out as an outputbased on the parameters set in the parameter setting device 5.

The calculation for correction of distortion in the geometrical positioncalculation device 6 can be performed suitably, for example, by makinguse of the following principle:

A circular image, which is formed on the surface of the image sensor 1by the fish-eye lens 2, is equivalent to an image projected on aspherical screen of a hemispherical body with a radius R centered on thefish-eye lens 2. Thus, desired correction of distortion can be made, ifthe spherical image is converted into a flat image. That is, as shown inFIG. 4, it is advisable if light is projected from the center of thehemisphere, O, onto a tangential plane P tangent to the sphericalsurface, and the resulting flat image can be displayed as an outputscreen after correction of distortion.

Here, moving the eyepoint is equal to moving the tangential plane P, asthe projection plane, on the spherical surface. The method of moving thetangential plane P comes in the following three types:

1) Rotation: Amount of rotation (α) about the eyepoint vector OO′ of thetangential plane P

2) Tilt: Amount of angular movement (β) in the vertical direction of theeyepoint vector OO′

3) Pan: Amount of angular movement (θ) in the horizontal direction ofthe eyepoint vector OO′

A computation concerning the correction of distortion (coordinatetransformation) is performed by the following procedure:

An XY coordinate system on the tangential plane P is transformed into anxyz coordinate system of the omnidirectional camera system, whereby thecircular image of the fish-eye lens 2 is converted into a flat image.

Concretely, as shown in FIGS. 5( a) to 5(e), the tangential plane P isscaled up or down, and rotated or rolled in accordance with thefollowing procedure, whereby its coordinate transformation is performed:

1) The barycenter O of the tangential plane P is placed at coordinates(R,0,0). At this time, the space coordinates of a point P₀ (X,Y) in thetangential plane P are (R,X,Y) (see FIG. 5( a)).

2) The tangential plane P is processed in accordance with a set zoommagnification to scale up or down the size of the tangential plane P to(1/zoom magnification) (see FIG. 5( b)).

3) The tangential plane P is rotated about the x-axis by α rad (rotationangle) from the y-axis toward the z-axis (see FIG. 5( c)).

4) The tangential plane P is rotated about the y-axis by (90°−β) rad(tilt angle) from the x-axis toward the z-axis (see FIG. 5( d)).

5) The tangential plane P is rotated about the z-axis by θ rad (panangle) from the x-axis toward the y-axis (see FIG. 5( e)).

If the destination of movement of the given point P₀ (X,Y) on thetangential plane P (see FIG. 4) is P₁(X₁,Y₁,Z₁) as a result of the aboveprocedure, P₁ is expressed by the following equation using a rotationmatrix:

$\begin{matrix}{\begin{bmatrix}X_{1} \\Y_{1} \\Z_{1}\end{bmatrix} = {\underset{PAN}{\underset{}{\begin{bmatrix}{\cos \; \theta} & {{- \sin}\; \theta} & 0 \\{\sin \; \theta} & {\cos \; \theta} & 0 \\0 & 0 & 1\end{bmatrix}}}\underset{TILT}{\underset{}{\begin{bmatrix}{\cos \; \left( {{90{^\circ}} - \beta} \right)} & 0 & {{- \sin}\; \left( {{90{^\circ}} - \beta} \right)} \\0 & 1 & 0 \\{\sin \left( {{90{^\circ}} - \beta} \right)} & 0 & {\cos \left( {{90{^\circ}} - \beta} \right)}\end{bmatrix}}}\underset{ROLL}{\underset{}{\begin{bmatrix}1 & 0 & 0 \\0 & {\cos \; \alpha} & {{- \sin}\; \alpha} \\0 & {\; {\sin \; \alpha}} & {\cos \; \alpha}\end{bmatrix}}}{\underset{Zoom}{\underset{}{\begin{bmatrix}1 & 0 & 0 \\0 & {1/{Zoom}} & 0 \\0 & 0 & {1/{Zoom}}\end{bmatrix}}}\begin{bmatrix}\begin{matrix}R \\X\end{matrix} \\Y\end{bmatrix}}}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$

The fish-eye lens 2 takes a surrounding 360-degree scene into itsfish-eye circle to form a fish-eye image. Generally, the fish-eye lens 2has image height characteristics, which are inherent distortioncharacteristics, with respect to an incident angle θ. That is, as shownin FIG. 6( a), the given point P₁ as a subject point on the tangentialplane P (see FIGS. 4 to 5( a)-5(e)) and the focused point Q of thefish-eye image in the fish-eye circle 2 a are correlated with each otherby the image height characteristics. Thus, a distance OQ in FIG. 6( a)is called an image height h, and this image height h takes a specificvalue with respect to the incident angle θ for each fish-eye lens 2 inaccordance with the distortion characteristics of the fish-eye lens 2.

FIG. 6( b) is an explanation drawing showing an example of the imageheight characteristics. As shown in this drawing, specifying theincident angle θ enables the image height h corresponding to theincident angle θ to be determined by a relevant image heightcharacteristics curve (a thick line in the drawing).

Here, as shown in FIG. 6( c), the distance OP₁ between the center O andthe point P₁ is given by the following equation:

OP ₁=√{square root over (X ₁ ² +Y ₁ ² +Z ₁ ²)}  [Equation 2]

Thus, the incident angle θ from the point P₁ (X₁,Y₁,Z₁) is given by thefollowing equation:

$\begin{matrix}{\theta = {{\arccos \frac{Z_{1}}{{OP}_{1}}} = {\arccos \frac{Z_{1}}{\sqrt{X_{1}^{2} + Y_{1}^{2} + Z_{1}^{2}}}}}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

As a result, the image height h can be found based on the incident angleθ obtained by the above equation and the image height characteristicsshown in FIG. 6( b).

Then, a point P₁′ shown in FIG. 6( c) (i.e., a point obtained byprojecting the point P₁ onto the xy-plane) is scaled up or down to theposition of the image height h to find the coordinates (x,y) of thefocused point Q shown in FIG. 6( a). Concretely, the coordinates of thefocused point Q are determined by the following equation usingOP₁′=OP₁·sin θ:

$\begin{matrix}{x = {{\frac{X_{1}h}{{OP}_{1}^{\prime}}\mspace{14mu} y} = \frac{Y_{1}h}{{OP}_{1}^{\prime}}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

In this manner, the coordinates (X₁,Y₁, Z₁) of the given point P₁ in thetangential plane P are transformed into the coordinates (x,y) of thefocused point Q upon correction of the distortion involved. That is, itbecomes possible to calculate the geometrical position of each pixel inthe input side image data (the pixel corresponding to the point Q on thecircular image), the pixel conformed to each pixel on the output screen(i.e., one corresponding to the given point on the tangential plane P).Here, the geometrical position in the present embodiment is found todecimal places.

An address table 10 shown in FIG. 1 stores table information which iscombined information correlating an input side address, as the addressof each pixel on the input side image data DI based on the calculationresults of the geometrical position calculation device 6, to an outputside address as a reference which is the address of each pixel on theoutput screen.

A matching sort unit 11 rearranges or sorts the output side addresses,stored in the address table 10, according to the respective input sideaddresses (a concrete method for this work will be described in detaillater).

A matching address table 12 stores table information which is combinedinformation correlating the output side address to the input sideaddress upon rearrangement in the matching sort unit 11.

FIG. 7 is an explanation drawing conceptually showing the relationshipamong the geometrical position calculation device 6, the address table10, the matching sort unit 11, and the matching address table 12. Asshown in the drawing, the geometrical position calculation device 6correlates the geometrical position of each pixel of the input sideimage data (xy plane) DI to the pixel on the output screen (uv plane) 20as a reference. As a result, in the address table 10, the input sideaddresses based on the geometrical positions are arrangedcorrespondingly in the sequence of the pixels on the output screen 20,namely, in the sequence of the output side addresses. In FIG. 7, acircular region 21 in the input side image data DI is a regioncorresponding to the visual field of the fish-eye lens 2, and afan-shaped region 22 in the circular region 21 is a cut-out region,namely, the region corresponding to the output screen 20.

The matching sort unit 11 rearranges the output side addresses in thesequence of the input side addresses. As a result, the output sideaddresses correlated to the input side addresses as a reference aftersorting are arranged in the matching address table 12.

FIGS. 8( a) to 8(g) are explanation drawings conceptually showing aconcrete example of processing by the matching sort unit 11. Thisexample illustrate a method called bit sorting, and shows a mode inwhich the input side address data of 5 bits initially arranged in thevertical direction in the drawing in the sequence shown in FIG. 8( a)(i.e., the data at the geometrical position) are rearranged in ascendingorder as shown in FIG. 8( f). With the present method, the state of “0”or “1” of the digits is detected, starting with a lower-order digit, andthe data are rearranged, as appropriate. That is, when the data in FIG.8( a) are rearranged in ascending order, with attention being focused onthe lowest-order bit, the data as in FIG. 8( b) are obtained. If thedata in this state are rearranged in ascending order, with the secondbit being focused on, the data as in FIG. 8( c) are obtained. Then, thesame procedure is repeated, with attention being paid to the third bit(FIG. 8( c)), the fourth bit (FIG. 8( d)), and the fifth bit (FIG. 8(e)), whereby sorting as shown in FIG. 8( f) can be finally carried out.

FIG. 8( g) is a block diagram showing an example of achieving theabove-described sorting by hardware configuration. As shown in thedrawing, all the above input side address data are initially writteninto an input buffer 31. That is, the state shown in FIG. 8( a) isformed. In this state, the lowest-order bit is first given attention. Ifit is “0”, a selector 32 writes the corresponding input side addressdata from the left side of an output buffer 33. If it is “1”, theselector 32 writes the corresponding input side address data from theright side of the output buffer 33. These actions are performedsequentially. As a result, a state corresponding to FIG. 8( b) is formedin the output buffer 33. In this state, the functions of the inputbuffer 31 and the output buffer 33 are reversed. Then, the second bit isgiven attention, and the same procedure is repeated to form a statecorresponding to FIG. 8( c). Afterwards, the same procedure is repeatedbetween the input buffer 31 and the output buffer 33, whereby the stateshown in FIG. 8( f) can be formed finally.

FIG. 9 shows the hardware configuration of FIG. 8( g) which is composedof 8 of the input buffers 31 and 8 of the output buffers 33. Such aparallel configuration enables bit sorting to be performed 4 bits at atime. Since the number of readings and writings is reduced to ¼, theprocessing speed can be quadrupled. Generally, the parallelconfiguration of 2^(n) of the input and output buffers 31 and 33 canincrease the processing speed n-fold. Thus, the numbers of the input andoutput buffers 31 and 33 may be determined in agreement with the desiredprocessing speed.

The address matching device 13 shown in FIG. 1 consists of an addressmatching unit 14 and an output side image data generation unit 15. Ofthese units, the address matching unit 14 checks the input side addressof each pixel of the input side image data DI, which has been sent outby the color processing unit 3 and loaded in real time, against theinput side address stored in the matching address table 12, and detectswhether they are coincident or not. When both input side addresses inthe address matching unit 14 are coincident, the output side image datageneration unit 15 combines the input side image data DI at the inputside address with the corresponding output side address to form outputside image data DO.

The output side image data DO formed in the output side image datageneration unit 15 is one obtained by eliminating the distortion of theinput side image data DI based on the information on the geometricalposition. The output side image data DO is sequentially stored in abuffer memory 8 as data which is a sequential combination of brightnessinformation and color information based on the input side image data DIfor each pixel. This combined data is reproduced, frame by frame, on thedisplay screen (output screen) 9 a of the display device 9 via aread-out circuit 16.

In the present embodiment, each time the results of the above checking(matching) are coincident, the output side image data DO is writtenrandomly into the buffer memory 8, with the output side address as areference. The readout of the output side image data DO is performedsequentially via the read-out circuit 16. A detailed description will beoffered later in connection with these points.

In this manner, the image in the predetermined cut-out region(fan-shaped region 22), the image corrected for distortion, isreproduced on the display screen 9a.

According to the present embodiment, the address matching device 13checks the input side address of each pixel of the input side image dataDI, which has been loaded in real time, against the input side addressstored in the matching address table 12. When both input side addressesare coincident, the input side image data DI at the coincident inputside address is combined with the corresponding output side address inthe output side image data generation unit 15 to form the output sideimage data DO.

In the present embodiment, the matching sort unit 11 is provided torearrange the output side addresses in the sequence of the input sideaddresses, so that the predetermined address matching in the addressmatching unit 14 can be carried out rationally. This is because theinput side addresses in the matching address table 12 are arranged inthe sequence of the input side image data DI inputted to the addressmatching unit 14, and single retrieval is enough to bring all the pixelsof the output side image data DO into correspondence with one pixel ofthe input side image data DI.

FIGS. 10( a) and 10(b) are explanation drawings conceptually showing thefurther detailed functions of the address matching device 13 and thebuffer memory 8. FIG. 10( a) shows the corresponding relationshipbetween the input side address (y,x) of the input side image data DIread out in the sequence of the addresses in the image sensor 1 and thecorresponding output side image data DO. On the other hand, FIG. 10( b)shows the matching action in the address matching unit 14, and the modeof generation of, and the manner of writing into the buffer memory 8 of,the output side image data DO in the output side image data generationunit 15.

As shown in FIG. 10( b), the address matching unit 14 checks the inputside address of the input side image data DI, sequentially read out inreal time, against the input side address (y,x) stored in the matchingaddress table 12. Since the input side address (y,x) in the matchingaddress table 12 in the present embodiment has a fractional part, ajudgment is made with its whole number part as a reference. Concretely,the contents of a line counter 41 representing the y-coordinate of theinput side address of the input side image data DI and the contents of acolumn counter 42 representing the x-coordinate thereof are compared.Here, the y-coordinate of the line counter 41 and the x-coordinate ofthe column counter 42 are given as integer values.

The output side image data generation unit 15 selects a total of 4pixels including the pixel at the coincident address and pixels in thevicinity of this pixel, namely, respective pixel data on the pixel atthe input side address (y,x), the input side address (y, x−1) adjacenton the same line to the pixel at the input side address (y,x), and theinput side addresses (y−1, x−1) and (y−1, x) adjacent one line ahead tothem, and forms each pixel data DO-1 for the output side image data DOfrom the total 4 pixels designated as 43. Thus, the output side imagedata generation unit 15 has a line buffer memory 44 for storing the oneline equivalent of data. In forming the each pixel data DO-1 as statedabove, brightness with respect to the adjacent pixel is found by linearinterpolation based on the value of the fractional part of the inputside address (y,x) in the matching address table 12. Here, theinterpolation need not be limited to linear interpolation, if it is aninterpolation weighted with the value of the fractional part. The eachpixel data DO-1 also contains color information generated based on thefour pixels 43, although this is not shown.

The each pixel data DO-1 formed in the output side image data generationunit 15 is written into the buffer memory 8. In this case, whenever theeach pixel data DO-1 is formed, it is written randomly into the buffermemory 8, with the output side address as a reference. Thus, the buffermemory 8 has the each pixel data DO-1 written thereinto in a state inwhich these data are arranged sequentially in the sequence of the outputaddresses. As a result, the readout of the each pixel data DO-1 isperformed sequentially, beginning at the start of the buffer memory 8.

Second Embodiment

In the first embodiment shown in FIG. 1, the each pixel data DO-1 arearranged in the sequence of the output side addresses in the buffermemory 8, as mentioned above. The configuration for readout of the eachpixel data DO-1 can be simplified accordingly.

In writing the each pixel data DO-1 formed in the output side image datageneration unit 15 into the buffer memory 8, the writing is carried outrandomly, with the output side addresses as a reference. In writing theeach pixel data DO-1 corresponding to the peripheral portion of thefan-shaped region 22 (see FIG. 10 (a)) with great distortion in theinput side image data DI, therefore, it becomes difficult to ensure asufficient writing speed, or in some cases, it becomes impossible tocarry out writing. The reason is that in the case of the each pixel dataDO-1 in a region corresponding to the above peripheral portion, the datamay be intermingled, and the writing interval may exceed the writingcapacity of the buffer memory 8, causing an overflow.

The present embodiment is designed to avoid such a writing disablingstate, and differs from the embodiment of FIG. 1 in the manner ofwriting into and reading from the buffer memory 8. That is, each timethe each pixel data DO-1 is formed, it is written sequentially into thebuffer memory 8, whereas the data is read out of the buffer memory 8randomly.

FIG. 11 is a block diagram showing an omnidirectional camera systemaccording to a second embodiment of the present invention. In thisdrawing, the same portions as those in FIG. 1 are assigned the samenumerals as those in FIG. 1, and duplicate explanations are omitted. Inthe present embodiment, output side image data DO formed by an outputside image data generation unit 25 are sequentially written into abuffer memory 8 each time they are formed. On the other hand, the outputside image data DO are read out randomly from the buffer memory 8 via aread-out circuit 26 by reference to a read-out address table 18, andreproduced on a display screen 9a of a display device 9.

The read-out address table 18 stores table information which is combinedinformation obtained by correlating a memory address of the buffermemory 8 to an output side address as a reference upon sorting by aread-out sort unit 17. The read-out sort unit 17 rearranges the memoryaddresses of the buffer memory 8 based on the output side addressesstored in a matching address table 12.

According to the present embodiment, writing of the output side imagedata DO into the buffer memory 8 is performed sequentially, so that anoverflow of written information as in random writing does not occur.Instead, the buffer memory 8 has the output side image data DO randomlywritten thereinto, so that the data need to be read out randomly in thesequence of the output side addresses. Information for such readout isin storage at the read-out address table 18. Thus, by reference to thecontents of storage in the read-out address table 18, random readout canbe carried out in the sequence of the output side addresses, asdetermined beforehand. This point will be described in further detailbased on FIG. 12.

FIG. 12 is an explanation drawing conceptually showing the relationshipamong the geometrical position calculation device, the address table,the sort unit, and the matching address table shown in FIG. 11. Thisdrawing corresponds to FIG. 7 in the first embodiment. Thus, the sameportions as those in FIG. 7 are assigned the same numerals as those inFIG. 7, and duplicate explanations are omitted.

As shown in FIG. 12, the aforementioned matching is carried out usingthe input side address table (y,x) of the matching address table 12. Onthe other hand, the read-out address table 18 is prepared using theoutput side address table (v, u) of the matching address table 12. Thatis, a comparative table containing the output side addresses of theoutput side address table (v, u) arranged in the sequence of an addressnumber table 19 of the buffer memory 8 is prepared. Then, a read-outsort unit 17 rearranges the address numbers in the comparative tableaccording to the output side addresses as a reference, whereby a tabledescribing the addresses of the buffer memory 8 corresponding to theoutput side addresses can be prepared. This table is stored in theread-out address table 18, and random readout based on this informationmakes it possible to reproduce the output side image data DO arrangedsequentially.

FIGS. 13( a) and 13(b) are explanation drawings conceptually showing thefunctions of the address matching device and the buffer memory shown inFIG. 11. This drawing corresponds to FIG. 10 in the first embodiment.Thus, the same portions as those in FIG. 10 are assigned the samenumerals as those in FIG. 10, and duplicate explanations are omitted.FIG. 13 shows that each time the each image data DO-1 is formed in theoutput side image data generation unit 15, it is sequentially writteninto the buffer memory 8, beginning with the start of the buffer memory8.

According to the present embodiment, as described above, loading intothe buffer memory 8 can be performed sequentially, so that an overflowof data when read in can be avoided. For readout, by contrast, data needto be read out randomly. In this case, however, the interval for readoutis constant, so that an overflow of information is not caused.

In the above first and second embodiments, the matching sort unit 11 isprovided to rearrange the output side addresses in the sequent of theinput side addresses, but sorting need not necessarily be performed.Although the number of retrievals for matching is increased, the presentinvention includes a case where no sorting is performed. That is, themere provision of the address table 10 and the address matching device13 makes it possible to construct a camera system rid of the framememory 4 (FIG. 14) used in the conventional technology, although theefficiency of matching is reduced.

Also, table information on a specific region is stored in an addresstable having the same functions as those of the address table 10.Moreover, the respective output side addresses are rearranged beforehandin correspondence with the respective input side addresses. By thesemeasures, the same table information as the sorted table informationstored in the matching address table 12 can be stored in connection withthe above specific region. In this case, therefore, rational matchingcomparable to address matching involving the matching sort unit 11 canbe performed in connection with the above-mentioned specific region.

If a plurality of the above specific regions are set, and sorted tableinformation as mentioned above is stored for each of the regions,rational matching can be performed in regard to each of the regions. Inthis case, control may be exercised such that the respective regions areautomatically switched as appropriate.

In the aforementioned first and second embodiments, moreover, the eachpixel data DO-1 for the output side image data DO is formed with the useof the pixels located one line ahead. Thus, the line buffer memory 44covering one line is provided. If only the input side address adjacenton the same line is utilized, however, the line buffer memory 44naturally becomes unnecessary. The provision of the line buffer memoriesenough for two lines or more, on the other hand, can form high accuracyoutput side image data DO utilizing information on correspondingly manyinput side addresses. Hence, the number of the line buffer memories maybe selected in consideration of the accuracy of a reproduced image.

The aforementioned first and second embodiments have the fish-eye lens 2as their lens. Thus, the calculation in the geometrical positioncalculation device 6 is designed to correct the distortion of the inputside image data DI by the fish-eye lens 2, but this processing is notlimitative. Processing for imparting a desired distortion todistortion-free input side image data DI taken in by an ordinary lens isalso included in the present invention. That is, the geometricalposition calculation means includes not only the correction ofdistortion of an image incorporated by the lens, but also processingsuch as correction for distorting an undistorted image. It is essentialthat no particular limitation be imposed on processing, if it transformsthe geometrical position of an image taken in by a lens.

1. An image processing apparatus, comprising: parameter setting meanswhich has a setting of a parameter concerned with at least one of pan,tilt, zoom and rotation for cutting out a region to be displayed in animage taken in by a lens; geometrical position calculation means forcalculating a geometrical position of each pixel in input side imagedata based on an output signal of imaging means, the each pixelcorresponding to each pixel on an output screen, in order to performpredetermined transformation of a geometrical position of an image inthe region based on the parameter; an address table for storing tableinformation which is combined information obtained by correlating aninput side address, as an address of the each pixel of the input sideimage data based on calculation results of the geometrical positioncalculation means, to an output side address as a reference which is anaddress of the each pixel on the output screen; and address matchingmeans which checks the input side address of the each pixel of the inputside image data, loaded in real time, against the input side addressstored in the address table, and when both input side addresses arecoincident, combines the input side image data at the input side addresswith the corresponding output side address to form output side imagedata, and also sends out the output side image data.
 2. A camera system,comprising: imaging means for forming a digital signal representing animage taken in by a lens; parameter setting means which has a setting ofa parameter concerned with at least one of pan, tilt, zoom and rotationfor cutting out a region to be displayed in the image; geometricalposition calculation means for calculating a geometrical position ofeach pixel in input side image data, the each pixel corresponding toeach pixel on an output screen, in order to perform predeterminedtransformation of a geometrical position of an image in the region basedon the parameter; an address table for storing table information whichis combined information obtained by correlating an input side address,as an address of the each pixel of the input side image data based oncalculation results of the geometrical position calculation means, to anoutput side address as a reference which is an address of the each pixelon the output screen; and address matching means which checks the inputside address of the each pixel of the input side image data, loaded inreal time, against the input side address stored in the address table,and when both input side addresses are coincident, combines the inputside image data at the input side address with the corresponding outputside address to form output side image data, and also sends out theoutput side image data.
 3. The camera system according to claim 2,wherein the address table stores table information in which the outputside addresses in the table information based on the calculation resultsof the geometrical position calculation means in connection with aspecific region are rearranged according to the input side addresses. 4.The camera system according to claim 3, wherein the address table storesa plurality of pieces of table information in which the output sideaddresses are rearranged according to the input side addresses inconnection with a plurality of the specific regions.
 5. A camera system,comprising: imaging means for forming a digital signal representing animage taken in by a lens; parameter setting means which has a setting ofa parameter concerned with at least one of pan, tilt, zoom and rotationfor cutting out a region to be displayed in the image; geometricalposition calculation means for calculating a geometrical position ofeach pixel in input side image data, the each pixel corresponding toeach pixel on an output screen, in order to perform predeterminedtransformation of a geometrical position of an image in the region basedon the parameter; an address table for storing table information whichis combined information obtained by correlating an input side address,as an address of the each pixel of the input side image data based oncalculation results of the geometrical position calculation means, to anoutput side address as a reference which is an address of the each pixelon the output screen; matching sort means for rearranging the outputside addresses stored in the address table according to the input sideaddresses; a matching address table for storing table information whichis combined information obtained by correlating the output sideaddresses to the input side addresses upon rearrangement by the matchingsort means; and address matching means which checks the input sideaddress of the each pixel of the input side image data, loaded in realtime, against the input side address stored in the matching addresstable, and when both input side addresses are coincident, combines theinput side image data at the input side address with the correspondingoutput side address to form output side image data, and also sends outthe output side image data.
 6. The camera system according to claim 2,wherein the geometrical position calculation means finds the input sideaddress to decimal places, and outputs the input side address as adecimal value, and the address matching means uses, as a reference, aspecific pixel corresponding to a whole number part of the input sideaddress in the input side image data, finds brightness of the specificpixel by interpolation for weighting with a value of a fractional partof the input side address based on brightness of a plurality of pixelsadjacent to the specific pixel, and takes the brightness of the specificpixel as brightness of a pixel in the output side image datacorresponding to the specific pixel.
 7. The camera system according toclaim 6, wherein the address matching means has a buffer memory forstoring at least one line equivalent of data, and is adapted to find thebrightness of the specific pixel belonging to a next line by use ofbrightness information on each pixel stored in the buffer memory.
 8. Thecamera system according to claim 2, further comprising a buffer memoryfor storing the output side image data, the buffer memory having theoutput side image data written randomly thereinto, with the output sideaddress as a reference, each time results of the checking arecoincident, and wherein readout of the output side image data isperformed sequentially.
 9. The camera system according to claim 2,further comprising a buffer memory for storing the output side imagedata, the buffer memory having the output side image data writtensequentially thereinto each time results of the checking are coincident,further comprising read-out sort means for rearranging memory addressesof the buffer memory according to the output side addresses, and aread-out address table for storing table information which is combinedinformation obtained by correlating the memory addresses to the outputside addresses as a reference upon rearrangement by the read-out sortmeans, and wherein readout of the output side image data is performedrandomly based on the table information of the read-out address table.10. The camera system according to claim 2, wherein the lens is afish-eye lens having a wide-angle visual field, and the transformationin the geometrical position calculation means is processing forcorrecting distortion of the image in the region.
 11. The camera systemaccording to claim 2, wherein the transformation in the geometricalposition calculation means is processing for distorting the image in theregion.